Avoiding selection bias : A unified treatment of thresholded data 
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When searching for populations of rare and/or weak signals in noisy data, it is common to use a detection 
threshold to remove marginal events which are unlikely to be the signals of interest; or a detector might have 
limited sensitivity, causing it to not detect some of the population. In both cases a selection of data has occurred, 
which can potentially bias any inferences drawn from the remaining data, and this effect must be corrected for. 
We show how the selection bias is naturally avoided by using the full information from the search, considering 
both the selected data and our ignorance of the data that are thrown away, and considering all relevant signal and 
noise models. This approach produces unbiased estimates of parameters even in the presence of false alarms 
and incomplete data. 



Introduction — There are many areas of the physical sci- 
ences where experiments or observations are performed in or- 
der to measure or search for populations of events or signals. 
For example, in astrophysics there are searches for Type la su- 
pernovae (SNIe) [1 |, gravitational-waves [2|, extra-solar plan- 
ets and high energy neutrinos 015]; and galaxy cluster sur- 
veys [6][7]. Often the aim is to use the resulting set of observa- 
tions to infer global parameters of interest, e.g. quantities that 
characterise the population distribution. For the case of SNIe 
for example, one may aim to infer cosmological parameters 
from luminosity distance and redshift measurements. In this 
paper we focus on these global quantities rather than proper- 
ties of individual events and show how to derive an unbiased 
inference from data sets that are subject to selection bias. 

Many experiments are subject to selection bias, which we 
define as the influence that a restriction on the measured data 
has on quantities inferred from a population of events or sig- 
nals. Data could be selected either deliberately by the exper- 
imenter, in order to reduce the background of noise events, 
or inadvertently through limitations of the detection equip- 
ment, such as a saturation point or limited sensitivity to weak 
signals. An example in the astrophysical literature is a flux- 
limited survey (SJ. We draw a careful distinction between se- 
lection bias and biases induced by an incorrect application of 
prior information |9l[T0), as the terms "Malmquist bias" ifTTT — 
[PD or "Lutz-Kelker bias" [ 14 1 are sometimes used to describe 
both types of bias. 

The problem of selection bias is present in many areas 
and in some fields is extensively documented. In astronomy, 
Kelly [15 1 takes a statistical approach to the bias problem 
which is closest to ours, considering both detected and non- 
detected events. In cosmology, the traditional approach has 
been to produce unbiased estimators of individual event pa- 
rameters [ 16 -20 1 which are then used as input for inference 
of global quantities. Thresholding can be applied to reduce 
the number of false alarms from a background population to a 
tolerable level, for example thresholding on radio pulse width 



is applied to data within the LOFAR NuMoon ETl project. 
The ANTARES [22 J neutrino experiment applies a thresh- 
old on a fitting parameter to optimise the detection efficiency 
of cosmic neutrinos whilst minimising the effect of the at- 
mospheric neutrino and muon background. In gravitational 
wave searches |231 . thresholds are set on the signal-to-noise- 
ratio (SNR) of an event to reduce the volume of data to be 
processed, and additional cuts and vetoes are applied within 
the data-analysis pipeline in order to minimise the false alarm 
rate due to detector noise. 

In this letter we apply Bayesian methods to provide a pre- 
scription for completely avoiding selection bias. It has been 
previously shown [15], for astronomical data-analysis, that to 
avoid bias in a signal dominated environment (i.e. no false 
alarms or no events due to noise alone) one should account 
for both the detections and the false dismissals within an ex- 
periment. We generalise and complete this approach by also 
considering the probabilities of detection and dismissal for 
all other relevant models, most importantly the noise model. 
Our approach correctly includes the information content of the 
data which are thrown away, making no assumption that they 
contain only noise, nor do we assume that detected events are 
produced only by true signals. 

We will first derive general expressions for the posterior 
probability density functions (pdfs) on the parameters of inter- 
est for any threshold-influenced analysis. We follow this with 
an example application to a simulated case of cosmological 
inference using standard candle transients. We conclude with 
a discussion of the uses and implications of this approach. 

General Formulation — We consider an experiment in 
which many measurements are made, each producing data 
which may contain potential detections, and which are then 
compared with a detection threshold. We refer to any mea- 
surements which pass the threshold as "triggers", since in 
most practical cases we will include one or more noise-only 
models alongside the true signal model. Any measurements 
which do not pass the threshold are discarded. We will derive 
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an expression for calculating global parameters based on the 
triggers, but which also takes into account the cases when a 
trigger was not produced. These non-trigger cases also carry 
information about the source population, and one cannot as- 
sume the lack of a trigger implies the lack of true signal, or 
vice-versa. 

We divide the experiment into multiple measurements, 
whose results we suppose to be statistically independent of 
one another. We will denote the outcome of the experiment 
D. The threshold is set on the measured quantity x, which 
we will call the "detection statistic". In general x may be 
a maximum-likelihood statistic or simply a direct measure- 
ment of a physical parameter value but in all circumstances 
this measurement will be subject to noise. For simplicity of 
notation we only consider situations in which values of x are 
restricted to be greater than a threshold jc tn E4ll 

If the statistic passes a threshold x > x tn a trigger is pro- 
duced, and the value of x is recorded; we denote this case 
Z) + , and D = (D + ,x). If on the other hand x < x tn , no trig- 
ger is produced and no value for x is recorded and D - D . 
We define {fi} as our set of plausible models and must in- 
clude at least two models with one describing the presence 
and the other the absence of a signal. In general the method 
extends to many alternative signal models. When specifying 
a set of models in this way we allow that any model may be 
true, and that the trigger has a non-zero probability under all 
of the models. For any single model H the likelihood func- 
tion defining the distribution of the statistic x is expressed as 
p(x\H,Qn,y,Pj where 6^ are the parameters unique to this 
model, and y are the global parameters which are common 
to all trials (following standard Bayesian notation we use / 
throughout to represent additional information regarding the 
experiment, and the global parameters y). Upon performing 
the single trial, we are interested in inferring the parameters 
y. In the case of a statistic above threshold, D + , we can write 
the posterior distribution on y as 

p(y\D + ,x,I) ^Pir.W*,*,!) 

trigger likelihood p{D + ,x\yj) 

P(j\i)' 



P(x\I) 



Y jP (x\y,'H,r)P('H\7,I) (1) 



where PifH\y, I) represents the prior probability on the model 
H conditional on the global parameters. Here we are sum- 
ming the trigger likelihoods over the possible models, and we 
do not assume that the model priors are independent of the 
global parameters, which will be important when discussing 
signal abundance and rates. We have used the fact that the 
probability of a trigger conditional on the value x is strictly 
equal to H(x - x tn ) (where H(x) is the Heaviside step func- 
tion) which is equal to unity for x > x&. If the model has local 
parameters On, we calculate the factor p(x\y, f H , /), i.e. the 
likelihood of the data conditional on the model and the global 



parameters, by marginalisation: 

p(x\y,<H,r) = J da H p(x\0< H ,y,<H,r)p(0 9{ \y,<H,I), (2) 

where is the local parameter space associated with the 
local parameter Q<n and the model Iri. Note that we do not 
modify this likelihood function to reflect our knowledge of 
the threshold. 

In the alternative case D where we discarded the specific 
value of x, we do still know that x < x±. For a non- trigger we 
can therefore write the posterior probability on y (analogous 
to Eq. ([]])) as 

non-trigger likelihood p{D~~ \y,I) 

P(7\D-,D = ^^YP(D\y,<Hj)P(<H\y,I) (3) 

As the value of x is unknown the posterior on y must be 
marginalised over both the local parameters 0^ and the un- 
known data x in the range x < x±. Therefore only the informa- 
tion D appears in the likelihood of a non-trigger conditional 
on the model and the global parameters: 



P{D-\y,<H,I) 



dx p(x\0<H, y, <H, I) P (0<H\y, < H, I) 



X<Xih 



(4) 

The quantity derived in Eq. Q is related to what is known 
as the selection function or detection efficiency, defined as the 
probability of a trigger given the true signal parameters for a 
given signal model. Here it appears in a modified form: the 
probability of a non-trigger given a set of global parameters 
and a model. It is only in this term that the detection threshold 
x t h actually appears. In practice this quantity (just like the 
detection efficiency) can be difficult to estimate in some cases. 

An ensemble of trials — Using Eqns. ([T]i and Q together 
with Bayes' theorem we are able to isolate the single trial 
likelihood functions (identified by the overbraces) in both the 
trigger and non-trigger cases, P(D + ,x\y, I) and p(D\y,I) re- 
spectively. We can now also consider analysing an ensem- 
ble of N trials denoted as {£)}, where we index each D, using 
i e (l,N). The posterior probability for the global parameters 
after N independent trials are performed is simply 



p(y\{D}J) 



P({D}\I) 



\\p{Di\y,I). 



(5) 



We can further decompose this expression into trigger D + and 
non-trigger D factors, 



p{y\{D},I) 



P(J\D 
P(\D}\I) 



Y[p(D\xj\y,Ij)\\P(D-\yJ k \ 



7=1 



k=\ 



(6) 

where j and k index the n triggers and the N-n non-triggers re- 
spectively, and Ij, 4 allow for different background informa- 
tion (such as time-varying detector sensitivity) in each mea- 
surement. 
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We now focus on the common case where there are only 
two possible models: the presence CH + ) or absence CH~) of a 
single signal per trial. Equation (|6]i becomes 



p{y\{D)J) 



P(j\i) 



P({D}\I) 

\~[ [ P {D\ Xj \y,<H\ Ij)PdH*\y, Ij) 

+ P (D + , Xj \r, <H~, Ij)P(<H-fy, Ij)] (7) 
Y][P(D\y,<H + J k )P(>H + \7Jk) 



./=! 



N-n 



k=\ 

+P(D-\y,<H-,I k )P(<H-\y,I k )]. 

To relate this to more standard statistical terms, consider the 
key components which make up the expression, each of which 
are in reference to single trials: 

• p(D + , Xj\y, < 7Y + , Ij) is the likelihood of producing a trig- 
ger with statistic value Xj when a signal is truly present. 

• p(D + , Xj\y, 'H '~, Ij) is the likelihood of producing a trig- 
ger with statistic value Xj from only the background dis- 
tribution. 

• P(D \y, ( H + , 4) is the likelihood that a true signal does 
not produce a trigger, i.e. the false dismissal likelihood 
(or 1 -detection efficiency). 

• P(D^\y,'H^,Ii l ) is the likelihood that no trigger is pro- 
duced when no signal is present, i.e. the true dismissal 
likelihood. 

The abundance or true rate of signal events is important in the 
two model case, when the two models are signal and back- 
ground. As the two models give a complete description of the 
data, we can write PCH + \y,I) = 1 - P{ < H~\y,P). The actual 
rate of signals, or a quantity from which it can be derived, 
must be included as one of the global parameters y, and if 
it is not known then it can be estimated from the ensemble 
of trials. This makes the prior probabilities for the models 
dependent on the global parameters y, so the joint posterior 
distribution for the rate will not in general be separable from 
the rest of global parameters. Alternatively, if the true rate of 
signals is known, it can be substituted into these expressions 
instead. Neglecting this effect is a potential source of bias in 
inference of both the rate and other global parameters. We 
will show a specific example of this calculation, including the 
rate estimate below. 

The key idea of our approach (in terms best suited to the two 
model problem) is to use the true and false dismissal prob- 
abilities to incorporate the information from the absence of 
triggers. This requires that these quantities be either calcu- 
lated or measured. Without these, any observations which can 
produce non-triggers risk being either biased or sub-optimal 
and may require ad-hoc corrections. It is only by acknowl- 
edging and accounting for ignorance that we can derive the 



correct results. Equation (|7]i contains all the parts necessary 
to infer global parameters without bias, making full use of the 
information that is kept, and accounting for that which is de- 
liberately (or necessarily) discarded. 

A standard candle example — To illustrate our method, we 
will consider a toy model in which a telescope performs a flux- 
limited survey for transient standard-candle events, much like 
SNIe surveys. Although our model is simple and lacks the 
detail of real observations, it will serve to show the salient 
points of the method. In the model we are interested in in- 
ferring the global parameters R, the astrophysical event rate 
per unit co-moving volume, Hq, the Hubble constant, and Q. m , 
the matter density, from simulated trials. We assume that the 
telescope is able to scan a patch of the sky a = 2x2 deg 2 in 
At - 60 seconds and that this constitutes a single trial. For 
each trial, the "data" comprises a measurement of the dis- 
tance modulus p and the redshift z of any event that passes 
the thresholds. In this case we apply thresholds on satisfying 
p < p th and z < z th where p th = 45 is the distance modulus 
threshold and z th = 2 is a redshift threshold. Our measure- 
ments of p and z are Gaussian random variables with standard 
deviations o - ^ = 0.5 and cr z = 0.02z respectively, i.e. the 
fixed distance modulus uncertainty which can be interpreted 
as either an intrinsic scatter in the standard candle luminos- 
ity or as a measurement error, and a redshift dependent z un- 
certainty. We suppose that events arise through either a sig- 
nal model < H + or through a noise model < H~ . According to 
the signal model we define the distance modulus of a source 
as p = 5 log 10 (a?L) + 25 where the luminosity distance is 
measured in Mpc and defined using = (1 + z)du with 
the co-moving radial distance dm = c C H(z')~ ] dz' with c 
as the speed of light. The Hubble parameter H(z) is given 
by H(z) = H Q y/Sl m (l + z) 3 + Qjt(l + z) 2 + C1a where for this 
example we assume a flat universe Q.^ = and fix dark en- 
ergy parameters wq = — l,w a = such that £2 a = 1 _ Qn- 
The model includes only one local parameter, zo, the true red- 
shift of each signal. Under the noise model, we assume that 
events are drawn from a unit variance Gaussian distribution 
with mean distance modulus p^- = 46.8 and can be inter- 
preted as arising from a redshift-independent detector noise in 
the telescope at the telescope flux limit. All data recorded 
above threshold due to the noise model will have a corre- 
sponding redshift value drawn from the redshift prior since 
we assume an existing host galaxy would be incorrectly iden- 
tified as the host. 

We choose a maximum redshift z max = 4 such that the 
cosmological volume enclosed is large enough that with 
a high confidence the observed comoving volume V = 

a Jo ^m( z ')/[(1 + z')H(z')]dz' contains all detectable sig- 
nals, i.e. those true signals that we would expect to pass 
the thresholds. Within this sphere we assume a uniform 
distribution of sources in co-moving volume giving a prior 
distribution on the observed redshift of any given source as 
p(z\y,I) = KdjJ[(z)/(l + z)H(z) where k is a constant of pro- 
portionality ensuring the correct normalisation of the prior 
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2 3 4 5 
R(Kr 5 Mpc~ 3 Yr 1 ) 

FIG. 1. (upper) The simulated data set consisting of 1 15 potential de- 
tections or triggers (those with errorbars) with magnitudes less than 
f/ h = 45 and redshifts less than z th = 2 (bounded by grey dashed 
lines) from a total of 131400 independent trials. Of those, 108 were 
signals and 7 were noise events. Those that did not pass the threshold 
are shown without error bars. For both triggers and non-triggers we 
further subdivide measurements into signal model < H + (black) and 
noise model f H~ (grey). The red curve represents the true relation 
between the redshift and distance modulus, (lower) Joint probability 
contours on the combined global parameters of the cosmological pa- 
rameters Ho and fl m and the astrophysical rate R. The true values of 
these parameters are indicated by the solid red lines. The contours 
represent regions of equal enclosed probability at values correspond- 
ing to 1,2 and 3 standard deviations. The black contours correspond 
to the application of our method accounting for both triggers and 
non-triggers and both the signal and noise models. The dashed grey 
contours represent results from an analysis accounting for triggers- 
only and signal model only. The dotted grey contours represent an 
analysis accounting for triggers-only but including both the signal 
and noise models. 



L dz p(z\y, /) = 1. We note that this prior is dependent 
on the global parameter Q. m and identical for both models. We 
simulate 1 year of trials assuming 360 observations per night 
with a true astrophysical event rate (per comoving volume per 
unit observed time) of R = 2.7 x 10~ 5 Mpc~ 3 yr~' with cos- 
mological parameters Hq = 72 km s _1 Mpc _1 and Q. m = 0.3. 
The triggers (in addition to non-triggers) are shown in the up- 
per plot of Fig. [T] where we reveal which are produced in the 
presence and absence of a true signal. 

First consider the probability of obtaining a non-trigger per 
trial under the assumption of the signal model ( H + , 

P{D-\y,<H\I) = 1 - fd/i jdz j dz rfn,z,Zo\y,1{*,l) 







z — 

J' 







p(zo\yJ) 



erfc 



erfc 



zo-z 



^2<7-2( Z0 )^ 



(8) 



The complementary quantity, the probability density associ- 
ated with detection for the signal model, for the /th trial is 



p{H h zj\y, K = 1 - Jdzo piUj, z h zo\y, K\ I) 



{Hj-^Zoif (zj-zof^ 



i , p(zo\r, i) 

= I dzo - — - exp 



Ina-^a-^zo) 



2(7-2 



2crKzo) 



(9) 



For the noise model we have similar expressions but in this 
case the distance modulus is independent of the redshift. For 
non-trigger and trigger cases under the noise model we have 

P{D-\y,<H-,I)=\- Jdfi Jdz Jdzo pQi, z,Zo\y,1T,T) 



( 



-\ erfc 

4 



m- 



th 



dzo p(zo\y, I) erfc 



zo-z 



th 



(10) 



z — 

pQijAjfaW-,!) = J dz p(Mj,z j ,zo\y,'H-,r, 
o 

-max 

e-^nhfMK r I ( Zj - z ) 2 \ P (zo\y,I) 



2ncr n 



2cr 2 (z ) / o~z(zo) 



(H) 



We define the model priors assuming the following limit. 
For observation lengths Af the probability of there being more 
than a single true signal event (above threshold or otherwise) 
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within the observable volume V is <k 1 . Since the occurrence 
of a signal event during a trial is governed by the Poisson 
distribution, the prior probability of the signal model is ac- 
curately approximated as PCH + \y,I) = RVAt. We consider 
our two models as exhaustive and as such the complemen- 
tary prior noise model probability is given by PCH\y,T) = 
1 - RVAt. 

As shown in the lower plots of Fig. [T] we are able to si- 
multaneously infer a joint posterior probability on the global 
parameters Hq, Q m and R. The model can be made arbitrarily 
more complicated by including any number of other unknown 
parameters, global or local, e.g. the intrinsic scatter in dis- 
tance modulus o~ ^ or additional cosmological parameters. 

Two additional results are included with the aim of high- 
lighting the biases that can occur when failing to incorporate 
all relevant information into an analysis. The first of these 
results corresponds to the naive case where only the triggers 
are considered and only the signal model is assumed. These 
results are calculated by evaluating Eq. [9] for each trigger 
and therefore ignores the information contained within both 
the model prior and the non-triggers. In this case the sub- 
sequent cosmological parameter estimates are highly bias to- 
wards small Hq and large Q. m values with R not estimated as 
it is present only in the model priors. This dominant cause 
of this bias is the truncation of the fi values by the thresh- 
old leading to a disproportionate number of triggers at higher 
redshift. Larger values of Q,„ lead to an increase in the down- 
ward curvature of the redshift-distance modulus relation better 
matching the data at higher redshifts whilst smaller Ho values 
oppose this effect at low redshifts. 

The second bias result corresponds to an analysis that con- 
siders only triggers but assumes both a signal and noise model. 
The resulting cosmological parameter estimates now incorpo- 
rate the information present within the model priors but ignore 
the information in the non-triggers and subsequently yield a 
significant under-estimate of Q m and over-estimate of R (the 
astrophysical rate estimate has become completely saturated 
at the upper prior boundary and is not shown in Fig. [T}. In 
these examples the non-trigger contribution to the result is 
dominated by the likelihood of obtaining a non-trigger from 
the noise model. For this case the missing component of the 
likelihood would primarilly act to shift estimates of Q„, and R 
towards larger and smaller values respectively. 

Discussion — Using a simple application of Bayesian infer- 
ence we are able to derive a general formalism for analysing 
data which have been subjected to a threshold, either from the 
nature of an instrument or artificial by design. The resulting 
expressions account for both the knowledge of and the igno- 
rance of the actual result of an experiment. The method suc- 
ceeds for any number of false alarms and reduces to known 
formulae in limiting cases such as high signal to noise ra- 
tio, or zero false alarm rate. We find that knowledge of the 
true rate or abundance of signals represents crucial informa- 
tion when making inferences with an ensemble of trials, and 
that it can therefore be inferred from them. The simplicity and 
consistency of the Bayesian approach requires no ad-hoc cor- 



rections to be applied, and each term follows naturally from 
careful consideration of the problem. Prior knowledge of sig- 
nal and noise models must be incorporated, but measured val- 
ues of efficiency and false alarm rates can be used when they 
are difficult to derive analytically. Our method has applica- 
tions in many noise-limited fields of research, ranging from 
particle physics to cosmology. It also allows one to study the 
change in precision of estimates when the threshold is lowered 
or raised, while retaining an unbiased result, and this could be 
useful when determining the optimal tradeoff between sensi- 
tivity and amount of data to retain. 

Acknowledgments — The authors are grateful to J. Clark, 
T. Dent, S. Fairhurst, I. Harrison, M. Hendry, D. Keitel, 
T. G. F. Li, W. Del Pozzo, G. Pratten, R. Prix, C. Rover, 
B. S. Sathyaprakash, P. Sutton, M. Vallisneri C. Van Den 
Broeck, and G. Woan for useful discussions and comments. 



* chris.messenger@astro.cf.ac.uk 
1 johnv@nikhef.nl 

[1] M. Hicken, W. M. Wood-Vasey, S. Blondin, P. Challis, S. Jha, 
P. L. Kelly, A. Rest, and R. P. Kirshner, The Astrophysical 
Journal, 700, 1097 (2009). 

[2] B. Abbott et al., Phys. Rev. D, 77, 062002 (2008). 

[3] S. Udry and N. C. Santos, Annual Review of Astronomy & As- 
trophysics, 45, 397 (2007). 

[4] M. Ageron et al, Nuclear Instruments and Methods in Physics 
Research A, 656, 11 (2011). 

[5] R. Abbasi et al, arXiv.org, astro-ph.IM (2010). 

[6] S. W. Allen, A. E. Evrard, and A. B. Mantz, Annual Review of 
Astronomy & Astrophysics, 49, 409 (201 1). 

[7] C. Cunha, Physical Review D, 79, 63009 (2009). 

[8] T. J. Loredo, Statistical Challenges in Modern Astronomy IV 
ASP Conference Series, 371, 121 (2007). 

[9] M. A. Hendry and J. F. L. Simmons, Astronomy and Astro- 
physics (ISSN 0004-6361), 237, 275 (1990). 
[10] T. J. Loredo and M. A. Hendry, "Bayesian multilevel modelling 
of cosmological populations," in Bayesian Methods in Cosmol- 
ogy, edited by Hobson, M. P., Jaffe, A. H, Liddle, A. R., Muke- 
herjee, P., & Parkinson, D. (Cambridge University Press, 2010) 
p. 245. 

[11] A. S. Eddington, Monthly Notices of the Royal Astronomical 

Society, 73, 359 (1913). 
[12] K. G. Malmquist, Medd. Lund. Astron. Obs., 20 (1920). 
[13] A. S. S. Eddington, Monthly Notices of the Royal Astronomical 

Society, 100, 354 (1940). 
[14] T. E. Lutz and D. H. Kelker, Publications of the Astronomical 

Society of the Pacific, 85, 573 (1973). 
[15] B. C. Kelly, Astrophysical Journal, 665, 1489 (2007.). 
[16] S. D. Landy and A. S. Szalay, Astrophysical Journal, 391, 494 

(1992). 

[17] A. Mantz, S. W. Allen, D. Rapetti, and H. Ebeling, Monthly 
Notices of the Royal Astronomical Society, 406, 1759 (2010). 

[18] M. A. Hendry and J. F. L. Simmons, Astrophysical Journal, 435, 
515 (1994). 

[19] J. A. Willick and M. A. Strauss, The Astrophysical Journal, 507, 
64 (1998). 

[20] D. Lynden-Bell, S. M. Faber, D. Burstein, R. L. Davies, 
A. Dressier, R. J. Terlevich, and G. Wegner, Astrophysical 



6 



Journal, 326, 19 (1988). 
[21] M. Mevius, S. Buitink, H. Falcke, J. Horandel, C. W. James, 
R. McFadden, O. Scholten, K. Singh, B. Stappers, and 
S. Ter Veen, Nuclear Instruments and Methods in Physics Re- 
search A, 662, 26 (2012). 



[22] S. Adrian-Martinez etal, arXiv.org, 1111, 3473 (2011). 

[23] J. Abadie etal., Physical Review D, 81, 102001 (2010). 

[24] The derivation is similar for the opposite case (x < x th ) where 
the detector becomes saturated beyond the threshold value and 
information regarding the exact value of the statistic is lost. 



